Skip to content

Material for Efficient Parallel IO course on 11th January 2022

License

Notifications You must be signed in to change notification settings

EPCCed/archer2-parallelIO-2022-01-11

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation






Efficient Parallel IO on ARCHER2

CC BY-NC-SA 4.0

David Henty EPCC: 11th January 2022 09:30 - 17:00 GMT, online

One of the greatest challenges to running parallel applications on large numbers of processors is how to handle file IO. Standard Unix IO routines are not designed with parallelism in mind, and IO overheads can grow to dominate the overall runtime. Parallel file systems are optimised for large volumes of data, but performance can be far from optimal if every process opens its own file or if all IO is funnelled through a single controller process.

This hands-on course explores a range of issues related to parallel IO. It uses ARCHER2 and its parallel Lustre file system as a platform for the exercises; however, almost all the IO concepts and performance considerations are applicable to any parallel system.

We will give a general overview of how parallel IO is implemented in MPI-IO as these are the routines ultimately used by higher-level libraries such as HDF5 and NetCDF. A good understanding of the performance characteristics of MPI-IO is therefore very useful in optimising the IO performance of most parallel applications.

The course does not teach the detailed syntax of the various parallel IO libraries, but the Fortran source code for the benchmarking application used in the practical sessions should be useful reference material.

Prerequisites

Prerequisites: The course assumes an understanding of basic MPI programming in C, C++ or Fortran. Knowledge of MPI derived datatypes would be useful but not essential.

Requirements

Participants must bring a laptop with a Mac, Linux, or Windows operating system (not a tablet, Chromebook, etc.) that they have administrative privileges on.

They are also required to abide by the ARCHER2 Code of Conduct.

Timetable (all times are in GMT)

Unless otherwise indicated all material is Copyright © EPCC, The University of Edinburgh, and is only made available for private study.

Tuesday 11th January

Exercise Material

Unless otherwise indicated all material is Copyright © EPCC, The University of Edinburgh, and is only made available for private study.

Here is the parallel IO exercise sheet.. As explained in the sheet, source code and instructions for the IO benchmark can be found at https://github.com/davidhenty/benchio/.


This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

CC BY-NC-SA 4.0

About

Material for Efficient Parallel IO course on 11th January 2022

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published